Just a natural observation and some ranting about how useless the SQL DB format really is!
Since I always planned to make the databases small on disk, use as less as memory as possible and even work on slower systems, the results are pretty darn impressive (compare that to a bloated, overkilled SQL setup where we're talking gigabytes of memory and diskspace needed as well as cpu server or even more of them).
Both SQL and SOTDS testing as shown was run on a Intel Core2 Quad CPU, Q6600, 2.40ghz and 6gb memory with 3.5 HDD drives (probably Green WD type).
For example, my test database is around 939mb in my own raw format (human readable, with some key characters in place), the resulting compressed database on disk is actually only 388mb on disk all finalized and ready to be searched within.
Now, the same exact file exported as CSV becomes 6GB (6 gigabytes!) and a SQLITE readymade DB ended up at 12gb.
What the hell? Are they insane!
Not to mention, searching in SQL LITE command tool version caused and immense disk grinding I have never seen the likes of. My SOTDS Search Engine does not that at all :-) !
The SOTDS Constructor will actually pre-calculate every search possible and remove duplicates of words, leaving only numbered indexes left. This is the reason why SOTDS is superior next to SQL format which naturally also prevent disk grinding and most likely gives better search speed than the commercialized and industrialized SQL does.
How can SQL (which was invented in 1974, 40 years ago with countless improvements and involvements of thousands of people since then) become worse than my SOTDS Project (1 man+3 years) ?!
Oh well.
|