`job.trace(iterator=True)` is not safe for `.decode()` / per-line processing #3200

stdedos · 2025-05-20T11:00:14Z

stdedos
May 20, 2025

According to https://python-gitlab.readthedocs.io/en/stable/gl_objects/pipelines_and_jobs.html#id6, the result of .trace(streamed=False) is/are "entirely stored in memory in this example".

I believe similar behavior can be achieved via iterator=True, though without callbacks.

The function

python-gitlab/gitlab/utils.py

Lines 69 to 89 in 5a4acab

    
           def response_content( 
        
               response: requests.Response, 
        
               streamed: bool, 
        
               action: Callable[[bytes], Any] | None, 
        
               chunk_size: int, 
        
               *, 
        
               iterator: bool, 
        
           ) -> bytes | Iterator[Any] | None: 
        
               if iterator: 
        
                   return response.iter_content(chunk_size=chunk_size) 
        
               if streamed is False: 
        
                   return response.content 
        
               if action is None: 
        
                   action = _StdoutStream() 
        
               for chunk in response.iter_content(chunk_size=chunk_size): 
        
                   if chunk: 
        
                       action(chunk) 
        
               return None

even though it uses response.iter_content, which could support decode_unicode=True (def iter_content(self, chunk_size=1, decode_unicode=False):), but there is currently no way to pass that downward.

The result is a binary stream, which is unworkable as-is for line-based operations - as I "expect"/process the Job Log as a line stream (e.g. match this, ignore that ...). This also mirrors "average Joe's processing" of a job log: Go through it line-by-line, search etc

Not a blocker - just wondering if there is a more elegant, in-library workaround (or solution I have missed).

... or "food for thought" if development is due to happen.

Navigation Menu

Repository navigation

`job.trace(iterator=True)` is not safe for `.decode()` / per-line processing #3200

Uh oh!

{{title}}

Uh oh!

Replies: 0 comments

Select a reply

Uh oh!

job.trace(iterator=True) is not safe for .decode() / per-line processing #3200

Uh oh!

stdedos May 20, 2025

Replies: 0 comments

`job.trace(iterator=True)` is not safe for `.decode()` / per-line processing #3200

stdedos
May 20, 2025