Welcome to a small overview of the Stone Oakvalley's Text Database Searcher (SOTDS) Suite.

Note: SOTDS logic is based on a static database concept, meaning you can not add items quickly,
or even after it was constructed and compiled. If you have a database already, or want to add
just an single item or change just 1 single character, you have to do so in the original .DAT
file and then re-construct the ENTIRE database again with the Constructor. 

It may seem harsh and useless, but the idea behind SOTDS was to create the fastest possible
search for collections, of any size, that would not change too much. Imagine a written book,
never to be changed by author, can be viewed as a static database. Or, imagine you have
gadzillion of files from an old computer system which you know is 100% complete,
that too can be called a static database.

Now, especially for old computer system and their files, example Commodore Amiga Oriented ones,
finding filenames can be a very tiresome process and the .ADF diskimages and archives contains
paths and filenames, never before been properly indexed perhaps, which ignited my idea for the
SOTDS concept. After many many years of hard coding it emerges itself as The SOTDS Suite!



******** SEE BOTTOM OF THIS README FILE FOR REVISION HISTORY **********


You can read more at the following posts:
-----------------------------------------
http://www.stone-oakvalley-studios.com/post.php?id=000402092014002909
http://www.stone-oakvalley-studios.com/post.php?id=000402092014001731
http://www.stone-oakvalley-studios.com/post.php?id=000402092014202159
http://www.stone-oakvalley-studios.com/post.php?id=000420092014230354
http://www.stone-oakvalley-studios.com/post.php?id=000405092014033947

or visit the main page for SOTDS
http://www.stone-oakvalley-studios.com/0004_01_dashboard_index.php


Quick introduction for the SOTDS .DAT format:
(example based on SOS_TEKNO_Filename+Filesize+Path+Location_Database.dat which are included as example in SOTDS Suite installer)

1: ABCD
2: tekno_games_011999_cd2.bin670047168tekno99 
3: / audiow32.dll 112128 
4: games/dos/ 1ral11.zip 6958637 
5: games/ burndemo.zip 7132257 
6: etc.....


1: ABCD refers to internal logic, known as PART A,B,C,D, where by default
   A: Source + Source Size + Source Name
   B: Path or Paths Name
   C: Filename
   D: Filesize   
   
2:  means start of new source (which can have many sub-entries, can be thought of as a "container").
   "tekno_games_011999_cd2.bin" is the actual CD Image file used to index paths, filenames and filesizes.
    means start of Source Size string (filesize of the actual CD image).
    means start of Source Name string (just diskname of CD image). 
   
3: Paths either "/" for root, "dos/" for single path, or "games/dos/" as several paths, always ending with "/". 
   If ABCD was specificed at first line of database, PART B cannot occur in length of 0. You have to add at least / as root)
   
   Then $0 again to split between end of path and beginning of filename
   Lastly $0 to split filename end, and filesize start.
   
4-5: Same thing as in Point 3, but showing several paths.   

6: Step 3 is repeated until end of contents of "container" was reached. Next container starts again from point 2:

   .DAT file does not have to end with any special tag.

It should be noted, that PART A,B,C,D,E,F,G,H,I,J,K,L,M,N,O has some strict rules of formatting regarding text and numerics only:
A: Source + Source Size + Source Name = TEXT/NUMERIC + NUMERIC ONLY + TEXT/NUMERIC
B: TextField1/    = TEXT/NUMERIC, but must end with / (can later be removed in search engine with INI flag)
C: TextField2     = TEXT/NUMERIC
D: Numeric Field1 = NUMERIC ONLY
E: TextField3     = TEXT/NUMERIC
F: TextField4     = TEXT/NUMERIC
G: TextField5     = TEXT/NUMERIC
H: Numeric Field2 = NUMERIC ONLY
I: TextField6     = TEXT/NUMERIC
J: TextField7     = TEXT/NUMERIC
K: Numeric Field3 = NUMERIC ONLY
L: Numeric Field4 = NUMERIC ONLY
M: Numeric Field5 = NUMERIC ONLY
N: TextField8     = TEXT/NUMERIC
O: TextField9     = TEXT/NUMERIC

The .DAT file should always indicate how many types of PARTS you are using. Any mix of ABCDEFGHIJKLMNO is allowed, such as:
ABCDEFGHIJK
ABCFGLM
ABHNO
ADO
ACDE
ACE
ADG
ABD
etc...

They should be in alphabetical order, so "DHBA" is not possible, should be "ABDH".

What you put in of text data is up to you, as long as you are aware of the NUMERIC ONLY places,
in which end, if TEXT was detected in NUMERIC ONLY parts, the Constructor might crash, and the
Search Engine give crazy results out, even crashes!

There are currently NO KNOWN limits of data lenghts or amount of any PART A (containers),
PART B, C, D, E, F and H entries, and I have successfully compiled databases with raw data over 1GB,
more or less 180million rows of data without any significant impact on Search Speed or excessive memory use (still below <700mb).

Constructing however, is always slow, even for tiny databases. But even if Constructing seems slow,
imagine it actually pre-calculates every word, and even partial matches in words and the fact that
EVERY available search term that would actually result in at least 1 results is processed during
that time, not to forget the initial SOTDS concept of static database logic and structure,
it doesn't look that bad after all, when search results and search speeds are calculated even before you can blink your eyes.



Sorry for the quick and dirty text manual, revision history and lame SOTDS DAT specification. A better colored PDF in future will
be made available when time permits!








:::: ABOUT SOTDS SEARCHER :::
-----------------------------
The Search Engine Software

The main tool to enable you to search for anything you want!

Make sure you download any of my ready-made databases, or simply use my Tools to start
indexing yourselves (currently only 1 tool is available for release).

Usage: Gui Driven, simply start SOTDS_Searcher.exe and select a database. The rest is pretty obvious, no tricks to learn :-)







::: ABOUT SOTDS CONSTRUCTOR :::
-----------------------------
Compiles, compresses and indexes a SOTDS .DAT file. Note, constructing is slow, could take
many hours to complete. 

About compression (which both impacts on constructing and search speed time, as well as diskspace needed).

0 = BriefLZ gives bigger files and slower search speeds (uncompression based). Compression is faster.
1 = ZIP gives fair filesizes and fast search speed (uncompression based). Compression is fairly fast.
2 = LZMA gives smallest filesizes, but medium search speed (uncompression based). Compression is very slow.

A quick test gives us an slight overview:
zip - 36mb database size: 50 shown / 56101 hits for: the in 0.03s (1.47s)
lzma,7z - 30mb database size: 50 shown / 56101 hits for: the in 0.04s (1.79s)
brieflz - 50mb database size: 50 shown / 56101 hits for: the in 0.08s (1.81s)

Based on my research, ZIP (1) was chosen as default, but you are free to try the other
compression methods naturally.


Instructions as follows:
------------------------
Usage: SOTDS_Constructor.exe <file.DAT> <compression-type*> <-hindex> <-debug>

Where <path/file.dat> is a SOTDS suitable formatted database
Where <compression-type> is either '0' = BriefLZ, '1' = ZIP, '2' = LZMA (7Z)
* can be omitted, thus ZIP Compression is default
Where <-hindex> will auto generate unique ID for each row for current
location processed. Note: <-hindex> will use PART H. If in use already,
the request will be ignore.
Where <Debug> will export PARTx.txt in current directory and
keep all files created in Generated_Output/

-h -help or ? displays this help

Example: SOTDS_Constructor.exe test.dat
Example: SOTDS_Constructor.exe c:\testing\test.dat
Example: SOTDS_Constructor.exe test.dat 1
Example: SOTDS_Constructor.exe test.dat 1 -debug
Example: SOTDS_Constructor.exe test.dat 1 -hindex -debug


 



 
::: ABOUT SOTDS TOOL #1 :::
-----------------------------
Index Disk Structure

Will simply index your selected drives (or a chosen path) and let you store paths, filenames, dates
and filesizes to produce a .DAT file (human readable with some special chars) for use with the
SOTDS Constructor, which after processing can be used by the SOTDS Searcher software.


Usage: Gui Driven, simply start Tool_01_SOTDS_Generator_Disk_Structure.exe and generate a database
which the SOTDS Constructor can compile, index and compress - ready for use with the SOTDS Searcher.


That's it for now.

Stone Oakvalley Studios (c) 2015
post@stone-oakvalley-studios.com
www.stone-oakvalley-studios.com







::: REVISION HISTORY SOTDS CONSTRUCTOR :::
------------------------------------------
v1.7 - 29 Apr 2016
	01: Fixed a stupid bug that detected PART J to be processed partly as PART I, causing a crash/dead-stuck neverending loop (PART J was introduced in v1.5)
    02: Fixed a stupid bug if wildcards were indexed and not 1 entry in the current part contained 3 chars minimum at all, the partial process is ignored now, before it simply crashed Constructor, assuming there was always 3 chars available.
    03: Improved merging of files bug (introduced in v1.6, but didn't work as expected for really large databases), fixed now!
    04: Added a command line flag, "-debug2" if entered, the attempted allocation of memoryblocks above 20mb will be logged. "-debug" function as before, but both of them cannot be used at the same time

                     

v1.6 - 14 Feb 2016  
    01: Added missing parts for -DEBUG command
    02: Changed merging of files code, now slightly faster in some cases.

	
v1.5 - 08 Feb 2016  
    01: Added support for part IJKLMNO, where KLM is strictly numeric, the IJNO = text/numeric 
    02: Compiled with PureBasic V5.41LTS just to make sure. No other changes adjusted. 1 bug fixed.
    03: SOTDS_Searcher has been updated to support the same new parts added, namely IJKLMNO 


v1.4 - 08 Nov 2015
	01: Compiled with PureBasic V5.40LTS just to make sure, several adjustment in code to adjust to the latest PB version.
	02: Changed #PB_PackerPlugin_BriefLZ     1651666042 to 4
	03: Changed #PB_PackerPlugin_Zip         8022384 to 1 
	04: Changed #PB_PackerPlugin_Lzma        1819962721 to 2


v1.3 - 07 Aug 2015 - Phew, that was a marathon of coding!
	01: Transformed PART B into regular PART C logic, since PATHS or SUBPATHS can anyway be searched upon in partial mode. Previous B1 and B2 is now only B2 code.
	02: Added also PART E,F,G,H to allow even more fields.  Part H is also forced to NUMERIC just for convenience, since its fast to search for numbers like PART D was.
	03: All related tools such as Constructor, Partial and Searcher along with some databases was also fixed, converted, as PART B is no longer splitted, but simple stored AS-IS with / intact!
	04: All GUI related code has been removed, as in GUI mode everything runs slower. Command line output is already informing about conversions, as well as log files.
	05: Fixed possible problems related to BAT file mergering and execution, could sometimes fail to create file with no error reported, but final database would be screwed somehow. Fixed Now
	06: Moved processing of Partial Word Generator to appear later in the code, to not interfere or run multiple times during main operation, but rather in sequence as oppose to parallell.
	07: A bunch of other duplicated code was trimmed and improved.
	08: Added extra flag "-HINDEX" which automatically generates an unique ID for each entry that belongs inside the same LOCATION container. This feature will use PART H. It will create an unique id by counting from 1 and up. If PART H is detected already in the database, HINDEX will not create any unique ID. In other terms, you can add your own counting ID for each location, but to save original DAT filesize, you can use this option instead. Once created, this feature can be used to re-create the entire contents of a PDF page, directory structures (that was originall .zip, .adf and so fourth). Further, any PARTIAL word generation of this PART H will be disabled to ensure a stable operation and smaller databases (not needed, as its function is isolated to showing an UNIQUE id group only) and finding partial hits for UNIQUE numbers shouldn't be needed (partial can be used on everything else anyway). This option are most useful if you have a database with directory/file/path/filesize structure. For PDF databases, PARTH should be used as pagenumber reference, and thus -HINDEX should not be used.
    09: Removed the need for generating and loading "CAL" calibration files during initial splitting. Done on-the-fly-to-disk now, with limited buffers exported (as before, only internal change).
	10: Cleaned up Console output and added colors and with minor layout changes.








::: REVISION HISTORY SOTDS SEARCHER :::
---------------------------------------
v2.7 - 22 Jun 2016 
	01: Replaced progress bars with something that do not flicker (seems to be related to latest PureBasic behaviour changes, was not like this before)
	02: Fixed a bug in the "APPLY FILTER" listings, wrong results came out either totally wrong or not at all due to missing +1 counting.
	03: Minor GUI/WINDOWS improvements especially during the APPLY FILTER window (cancel, after found hits and more)


v2.6 - 11 May 2016 
	01: Added option to select a .ZIP downloaded database from my website and add into the SOTDS Suite directory automatically (instead of user having to deal with that)
        Makes it simpler to use my databases. The button is available during the "Choose SOTDS Database" popup before the search engine is initiated. After install, it will be added
        in the listview, ready to be loaded

		
v2.5 - 01/22 Apr 2016 
    01: Fixed an export bug for Extensions style HTML. Showed same entry twice in output.
    02: Removed limitation of one-instance of SOTDS using the same database
	03: Improved Extension search when typing instead of choosing from pulldown
    04: Compiled with PB5.42 LTS

	
v2.4 - 08 Feb 2016 
    01: Added support for part IJKLMNO, where KLM is strictly numeric, the IJNO = text/numeric. It means more fields are supported in entire SOTDS Suite.
    02: Tweaked some macros and added some more internally to make code cleaner and future adjustable.
    03: Reduced memory usage during startup, previously 1 million items in memory for each part were pre-allocated. This was probably overkill, so adjusted down to 10000, with automatic increase of 30000 each time until results neeeded.
    04: Minor GUI fixes and improvements, some bugs probably squeezed since V2.3 due to more macro coding.
    05: It should be noted that databases that uses the entire range of part A1A2A3BCDEFGHIJKLMNO (17 fields) can cause memory requirements increase and disk space needed for final database is increased including slower GUI updates.
    06: "Export CSV (TAB)" "Export HTML table" and "Export All Locations" where all put into a pulldown menu to save some space. Upon selection if works are before.
        Since I grouped the choices above into a pulldown menu, I had space for a filter feature. It will allow you to filter the entire current search hits found to add the logic of "MUST CONTAIN" for each available part.
        The new Filter Mode will remove permantly a complete line of fields in memory, so use with caution. If you need to get everything back, just do a search again and re-do the filtering, no problem.
        The new Filter Mode a little bit slower (just like Sort By Column function is), but has been fairly optimized to be useful at least. A progress bar was added too.
    07: "Export By Location" changed to "Export By (ID)", which makes more sense since its only active for UNIQUE_ID_MODE or SENTENCE_MODE enabled databases 
    08: Compiled with PureBasic V5.41LTS Final Release just to make sure.  


v2.3 - 08 Nov 2015
	01: Compiled with PureBasic V5.40LTS just to make sure, several adjustment in code to adjust to the latest PB version.
	02: Changed #PB_PackerPlugin_BriefLZ     1651666042 to 4
	03: Changed #PB_PackerPlugin_Zip         8022384 to 1 
	04: Changed #PB_PackerPlugin_Lzma        1819962721 to 2


v2.2 - 07 Aug 2015 - Phew, that was a marathon of coding!
	01: Removed PART B split path logic, now PART B acts the same as PART C (filename), ergo, full path is indexed. 
    02: Minimized duplicated codeblocks (added macros), so search speed is increased and more expandable in future for adding more parts and ensures safer more stable code allover.
    03: Maintained some minor bugs and prevents some possible crashes and/or memory leaks.
    04: Added support For PART E,F,G,H, where H is NUMERIC only (like Part D always was)
    05: Fixed minor bugs in export functions and rewrote the entire logic to ensure proper output (CSV/HTML)
    06: Remove support for Path_Char in INI file, not needed now as PART B had changed logic. Constructor and databases updated too.
    07: Removed support for HH:MM:SS conversion in both search engine and INI file, not needed anymore as by adding more parts, it would only slow down search engine (introduced for 3 weeks in v2.1)
    08: Approx 200kb of written code was removed and replaced by macro and other tricks to ensure faster and more stable operation. SourceCode is now approx 526kb as of this release.
    09: Added "Sentence Mode" to INI (values from 2 to max 5 allowed) and updated some databases to support it (RAWTEXT and PDF based ones), which utilize a 2,3,4 or 5 word sentence logic, so its more helpful than before with just a single word given back. No major code added to support this, its a simple organizing of columns locked to PART BCEFG and some header coloring to indicate which position the word was found in. Basically its a "first word", "last word", "middle word", "left middle word", "right middle word" at max. Another neat "side effect" of this search mode is to search for a page number (in PDF databases) which would actually represent the entire page of text orginally extracted from that page.
	10: Added "Export By Location", which is an incredible new feature that allows you to export the entire contents (as found) grouped by location name (which are useally unique). It will therefore create a nice HTML output style (with your selected locations to export), thus enabling to for instance export an entire page of text as found originally in the PDF during constructing. In other solutions, it will export the entire contents of a zip file, or Amiga ADF disk image or similar entry that contained more than just one row of data, simply because it would group by location name and its unique ID. This is a simple, effective but cool feature to know more about the location you found a interresting searched-for-word. It's pretty easy to export, simply do a query in the tab that has " (ID) " (in the databases that support it), choose the locations you want and all things are exported nicely into a block-html style
	11: If databases support the new PARTH (ID) mode, you can click on a selected search hit result with right mouse button, to get a menu with "Inspect ID", which will simply switch to the "Unique (ID)" tab, insert the number as selected  and perform a search. (so you dont have to memorize an Unique ID number for instance). The menu will not be available unless the databases has either Sentence Mode on and/or Unique ID (PartH) configured already.
    12: Various minor gui changes and info added here and there.
    13: Added button to configure/assign an "Local Access" path during the load database selection screen. It means you can assign a prefix path, like "C:\MyTOSEC-Archive\". During searching for filenames, this path will be added to the internal working, which means if you double click on a search hit (path or direct to a filename), it will either use Total Commander (if installed) or file explorer to open up the location of the file and thus you can access it locally and convenient. It should be noted that NONE of my databases support this, as my released databases are either custom-made, or slightly modified directory structure that does not exists anywhere else out there. This option is primarly for those who actually use my SOTDS Constructor tool to index their own harddrives and collections. If you are still old-school, you can modify the .INI entry "Local Access" yourselves too!
    14: Overall performance in search speed has been detected ONCE again :-)




v2.1 - 10 Jul 2015 
	01: Added auto-adjustment to allow all 50 items in list at the same time. In full mode, window requirements are 1280x1024 or more (1280x965), in small mode at least 1280x590
	02: Added "Load New Database" button, which bascially just exists, create a bat file and starts a new popup :-) Simple, and effective. 
	03: New flag feature added in Searcher/INI file to convert numeric values into hh:mm:ss time format. Usefull for song databases. Value is stored as seconds, but converted into appropriate format as specified in INI flag (if exists)
		Note: an example flag reads: "Seconds as = AD|HH:MM:SS", where we tell search engine to convert Part A,D,H to Hour:Minute:Seconds (when found). Does not work for any other PARTS as converting a text word into seconds are pointless :-)
		Possible combinations are any mix of "HH:MM:SS" in any order or style, like "MM" only, or "HH:SS" and so fourth.
		Features above added as result of including SOASC= and SOAMC= MP3 databases for release during early July 2015 (with re-release in August 2015).
	04: Adjusted the search speed timer to show 6 digits instead of 2.
	05: Adjusted and cleanup for the "About SOTDS" HTML (Installed Database output), visual fix only
	06: Added missing EXPORT CSV/HTML for with codestyles that has no A2, A3 in them, like CXD1A1, A1, C1D1A1 etc (produced by Tool_01_Generator_Disk_Structure.exe)

	






	
::: REVISION HISTORY SOTDS Generator Tool #1 (Index Disk Structure) :::
-----------------------------------------------------------------------
v1.5 - 08 Nov 2015
	01: Compiled with PureBasic V5.40LTS just to make sure, several adjustment in code to adjust to the latest PB version.

	
v1.4 - 07 Aug 2015
	01: Reworked code as PART B (paths are no longer appear in split form),added use of other PARTS instead and remove PATH CHAR in INI file export
	02: Instead of PART ABCD beeing used, the new structure is ABCDE
	A = FULL/LOCATION/PATH        (like  c:\windows\)
	B = Filedate as DD MMM YYYY   (like 27 jul 2015)
	C = Filename                  (like booter.dll)
	D = Filesize                  (like 49152)
	E = Filetime as HH:MM         (15:34) - 24hour clock
	Or without Datestamp, only used parts are ACD with no date and filetime naturally.


v1.2 - 10 Jul 2015
	01: Added option to select a directory instead of entire disk. Only one directory can be chosen, but everything within it will be indexed.
		This is just an quick update, maybe in future, a selection for adding other directories on a list might be added as feature.
	02: Fixed a text bug where driveletter appeared in uppercase, now fixed to lowercase (needed by the entire SOTDS Suite)
	03: Added question to construct and index database created during runtime to prepare it for searching for SOTDS Searcher


