I have a large CSV with millions of rows, how to use it in ESB

I have a large CSV with millions of rows, how to use it in ESB
Id like to import it into ESB so the end user can search the records.
From what I have read here and after attempting to import. It was
taking way to long.

Is there a work around for this other than the import option?

James
 

TigerSoftware

Administrator
Staff member
Hi James,

How long should "millions" of rows take to import?

Don't know what is in the data nor how many "millions" of rows we are talking about here.

There is no workaround when it comes to importing from a spreadsheet.

Seems like a rather high number of records to import at once. Besides the structure of your program and possible computer limitations. RAM, Processing speed, data storage...

I wouldn't recommend doing this to be honest.

I would try to break it up into smaller chunks/spreadsheets and import them one at a time.

Do remember that this amount of data will increase the size of the file you deliver to your customers. Can be quite a big size depending on the type of data you are importing. Ask Robbie about images. lol


Thomas
 
Last edited:

Mel Bel

Super Moderator
Staff member
That's too much data for most desktop software to handle, not just ESB software. The computer resources used just to load all of that data each time you start the software will result in wonky behavior and be a nightmare to deal with generally.

I put ESB 3 through some serious CSV import stress tests in the past and "millions" is not happening (even if each row has little data).

But it does a pretty good job of handling above 1 million rows in a single software build (even with several columns of data), but that's with different grid objects created (for 1 million rows of imported data, for example, you would need to create 10 or more grid objects with maybe 100,000 rows maximum of data imported per grid object).

To import faster you would need to split the "millions" CSV file into multiple CSV files of 5000 rows each that you import one at a time - I've actually done this and have tried importing many different size CSVs, and importing 3000 - 5000 rows at a time is the fastest, unless you have a crazy amount of data in the rows which will still make it a slow import if that's the case.

To create this correctly so that it is useful, you would need to have the CSV files split and categorized in some meaningful way and then create your grid objects around those categorizations. Then do all of your importing into each grid object and create the build.

Bottom line: what you are wanting to do won't be easy, and it will take a lot of effort to import the data and you'll need to create the software build a certain way for it to work acceptably, and that's just for roughly 1 million rows of data maximum imported per software build (and no excessive data overall in the CSV files you're importing; otherwise it will be even tougher to accomplish and it will still be a strain on resources).

Mel
 

TigerSoftware

Administrator
Staff member
That's too much data for most desktop software to handle, not just ESB software. The computer resources used just to load all of that data each time you start the software will result in wonky behavior and be a nightmare to deal with generally.


Mel

Not to forget our customers will have different computer configurations. Even if it worked on our computer, it may not work on their computer.

Not sure why anyone would need that amount of data for a desktop program like ESB.


Thomas
 
True not sure how long it should take...lol white some time. Yes I saw Mel's post which referred to The images issue as well. I may have a work around. If I use a spreadsheet I can look it with a password. The last thing would be in the Model Editor... Can I...

1. Disable exporting
2. Disable printing.

If so can you instruct me how?
 

TigerSoftware

Administrator
Staff member
ESB programs were never meant to be read-only.

I'm assuming you only want users to be able to access the data.


Thomas
 
Last edited:
Sorry you were correct Thomas I miss read you statement. I want the data to be read only whether that is in a spreadsheet that I pull in with the spreadsheet object or
 

TigerSoftware

Administrator
Staff member
Sorry you were correct Thomas I miss read you statement. I want the data to be read only whether that is in a spreadsheet that I pull in with the spreadsheet object or

Yeah, like I said, ESB wasn't created for that type of purpose.

A lot of things would need to be updated to make this work. Besides the forms itself, there is also reports, dashboards, spreadsheets, documents and other forms that would need to be set to read-only.

And how would updates work if we wanted to add different data?

From my current perspective, I don't really see a good enough reason to put my time into it. Compared to the stuff I am working on now like web automation and accessing webservices which will be more widely used and allow us to tap into larger markets.

I just don't think the majority of people would be using it as a read-only data source. In all the years I've been working on ESB, this is the first time anyone brought up the idea.


Thomas
 
Last edited:
Top