Loading analytics data with the TWA web GUI works really pleasantly for datasets of say 50MB/100K rows (completes quickly, doesn't hang/crash, gives timely/meaningful feedback etc).
When you're loading a data set of 100GB things are not so pleasant. The UI stops being your friend and the workarounds of loading the files directly into the TWA storage aren't much better (uses up 100's of GB of storage on the host OS which is a pain with virtualisation/networking, gives feedback only when you go diving through the API).
What I'm suggesting is this:
Create UI and services to allow data (in the current import format but also adding the attached example) to be loaded from network storage or databases e.g. SQL Query, FTP, SMB (because we shouldn't be fiddling with the files on the OS drive of an application server)
Infer data types from a subset of the file to be loaded e.g. only use the first 100K rows to inter data types as it may take 12+ hours to scan through the entire file or query result.
Provide status of the load
Is it working?
Where's it up to?
Can I stop it?
Can I see a preview of the sort of data being loaded?