Pipable - Alternatives to stdin/out
Posted: Tue Dec 13, 2016 6:05 pm
I love the pipable wim concept, and the ability to capture/apply via STDOUT/STDIN works for a number of use cases. Would you consider adding the ability to work with the pipable data via an API callback? This is a pattern I've seen in compression libraries that could be a be a way to allow preprocessing activities relevant to pipable wims:
Chunk
Encrypt
Split
etc.
Tools that work with STDIO (wget/wput) are great for generic http storage mediums. However some cloud storage providers may have unique constraints that require preprocessing data prior to passing it to their API/CLI. For example, they may have blob size limitations (or upload timeouts for a given chunk size). A callback could allow splitting/joining and other needs. In other words, sometimes a translator is needed between the two ends of the pipe, and as long as the consumer provides valid PWM data back to wimlib it shouldn't matter that it may have been sitting on cloud storage encrypted in 100 MiB chunks.
Some implementation ideas:
Opt-in flag for performance reasons
Stream chunk size param (so the callback can receive consistent chunks until empty)
Callback position relative to existing STDIN/OUT? Before/After?
The U-Stream library seems relevant to cloning/redirection of STDIO to C-style callbacks.
Don
Chunk
Encrypt
Split
etc.
Tools that work with STDIO (wget/wput) are great for generic http storage mediums. However some cloud storage providers may have unique constraints that require preprocessing data prior to passing it to their API/CLI. For example, they may have blob size limitations (or upload timeouts for a given chunk size). A callback could allow splitting/joining and other needs. In other words, sometimes a translator is needed between the two ends of the pipe, and as long as the consumer provides valid PWM data back to wimlib it shouldn't matter that it may have been sitting on cloud storage encrypted in 100 MiB chunks.
Some implementation ideas:
Opt-in flag for performance reasons
Stream chunk size param (so the callback can receive consistent chunks until empty)
Callback position relative to existing STDIN/OUT? Before/After?
The U-Stream library seems relevant to cloning/redirection of STDIO to C-style callbacks.
Don