Gigablast uses its own format for logging messages, for example,
1091228736104 0 Gigablast Version 1.234
1091228736104 0 Allocated 435333 bytes for thread stacks.
1091228736104 0 Failed to alloc 360000 bytes.
1091228736104 0 Failed to intersect lists. Out of memory.
1091228736104 0 Too many words. Query truncated.
1091228736104 0 GET http://hohum.com/foobar.html
1091228736104 0 http://hohum.com/foobar.html ip=4.5.6.7 : Success
1091228736104 0 Skipping xxx.com, would hammer IP.
The first field, a large number, is the time in milliseconds since the epoch. This timestamp is useful for evaluating performance.
The second field, a 0 in the above example, is the hostId (from hosts.conf) of the host that logged the message.
The last field, is the message itself.
You can turn many messages on and off by using the Log Controls.
The same parameters on the Log Controls page can be adjusted in the gb.conf file.
<Last Updated Sep 2014>
Gigablast is a fairly sophisticated database that has a few things you can tweak to increase query performance or indexing performance.
General Optimizations:
- Ensure that all drives can operate at maximum performance at the same time. Nowadays, the PCI-E bus could be the limiting factor, so be aware of that. Ensure you get maximum throughput and lots of disk seeks per second. By doing a cat /proc/scsi/scsi you can get info about your drives.
- Prevent Linux from unnecessary swapping. Linux will often swap out
Gigablast pages to satisfy Linux's disk cache. By using the swapoff command
to turn off swap you can increase performance, but if the computer runs out
of memory it will start killing processes without giving them a chance
to save their data.
- Consider adding more servers to your architecture. You can add the new servers to the hosts.conf file and Gigablast can rebalance the shards while still serving queries.
- Run one gb process per physical core, not including hyperthreaded cores. Hypthreaded cores are not that good.
Query Optimizations:
- Disable site clustering and dup removal. By specifying &sc=0&dr=0
in your query's URL you ensure that these two services are avoided and no
cluster records are loaded. You can also turn them off by default (site cluster by default and dedup results by default) on the
Search Controls page. But if someone explicitly
specifies &sc=1 or &dr=1 in their query URL then they will
override that switch.
- Disable gigabit generation. If accessing from the API set &dsrt=0 otherwise set results to scan for gigabits generation by dfefault to 0 in the search controls page for your collection.
- If you see lots of long black lines on the Performance graph then that means your disk is slowing everything down. Make sure that if you are doing realtime queries that you do not have too many big posdb (index) files. If you tight merge everything it should fix that problem. Otherwise, consider getting a raid level 0 and faster disks. Get SSDs. Perhaps the filesystem is severely fragmented. Or maybe your query traffic is repetetive. If the queries are sorted alphabetically, or you have many duplicate queries, then most of the workload might be falling on one particular host in the network, thus bottle-necking everything.
- If you see long purple lines in the Performance graph when Gigablast is doing slow query handling then that means Gigablast is distributed over a slow network.
Spidering and Indexing Optimizations:
- Disable link voting or link spam checking in the spider controls if you do not care about it. This is also useful when doing millions of injections and doing an indexdb rebuild using the rebuild tool later to pick up the link text.
- Disable dup checking. Gigablast will not allow any duplicate pages
from the same domain into the index when this is enabled. This means that
Gigablast must do about one disk seek for every URL indexed to verify it is
not a duplicate.
See the deduping enabled parm on the Spider Controls page.
- Enable use IfModifiedSince. This tells the spider not to do
anything if it finds that a page being reindexed is unchanged since the last
time it was indexed. Some web servers do not support the IfModifiedSince tag,
so Gigablast will compare the old page with the new one to see if anything
changed. This backup method is not quite as efficient as the first,
but it can still save ample disk resources.
- Disable make image thumbnails in the spider controls if you have not already.
- In the current spider queue, what are the statuses of each url? If the spider is always bottlenecking on adding links
that is because it does a dns lookup on each link if it has not previously encountered that subdomain. Otherwise, the subdomain IP when first encountered is stored in tagdb in the firstIp field. You might try using more DNSes (add more in the Master Controls).
- Ensure that the maximum spiders in the Master Controls and in your collection's Spider Controls is high enough.
|