Frequently Asked Questions
Frequently Asked Questions
IDOL - Backup, Restore and Combined Output During Initial Indexing
Posted by Kimmo Pyhältö (M-Files) on 30 September 2019 02:10 PM

During the initial indexing process, the whole material of the vault is indexed. That might take several weeks. If the indexing fails for some reason and there are no any backups, the whole indexing process might have to be done again.

This document describes how to take backups incrementally during the initial indexing process. Additionally it describes, how to change the indexing mode into Combined Mode, which provides further ruggedness to the process.


1. Taking an incremental backups during initial indexing

1.1 Considerations

Bakcups are taken separately from each engine. Plan the backup interval so its work effort is in line with the overall indexed mass and the time needed to index the whole mass. For example if the whole indexing takes three weeks, it would brobably be good to take six backup sets..

1.2 Process

1. Stop the M-Files Indexing Services (MFIndexingManager). Note that this stops all indexing so new material will not be added during the stop. It will be added afterwards after the indexing service starts.

2. Copy index log files IndexFLog.log (FileData) and IndexMLog.log (MetaData) to the safe place. They are located in M-Files Servers index folder, usually at C:\Program Files\M-Files\Server Vaults\Indexes\Combined\M-Files.

3. Wait until all IDOL indexing queues are empty.

- IDOL indexing queue location in each engine (example): I:\Single24\IDOLServer\IDOL\content\status

(- M-Files indexing queue location (example): C:\Program Files\M-Files\Server Vaults\Sample Vault\FileData\FileData\Temp\FIX\C_B91D75EF-A924-4583-A7BA-2A8F3C5614C3)

4. Take the backup from each backend engine and from Daily (might not create anything if it is empty) by opening browser in the frontend server and using following command. Note if you have not deleted the previously created backup files, you have to use a new name e.g. BU2, since the process cannot overwrite.

http://[backend server IP]:[engine port]/DREEXPORTIDX?FileName=[optional path and backup filename]

Example: http://192.168.75.130:10001/DREEXPORTIDX?FileName=BU

This will generate a file called [filename]-0.idx.gz into the engine's folder I:\Single24\IDOLServer\IDOL\content. The file includes all index data to the point of last full indexing batch.

Optionally, if the index is HUGE (more than 20 million objects), you can export only a part of it by using command MinDate and MaxDate. 

Example: http://192.168.75.130:10001/DREEXPORTIDX?FileName=BU010113_010313&MinDate=01/01/2013&MaxDate=01/03/2013

That would export documents that are indexed between 01/01/2013 and 01/03/2013 would be exported. Make sure that you name the files accordingly since in this method you will have several backup files from each engine.

You can use a Power Shell script like below to run multiple backups simultaneously.

$navOpenInBackgroundTab = 0x1000;
$ie = new-object -com InternetExplorer.Application
$ie.Navigate2("http://192.168.75.128:9001/DREEXPORTIDX?FileName=BU");
# backup from Daily
$ie.Navigate2("http://192.168.75.130:10001/DREEXPORTIDX?FileName=BU", $navOpenInBackgroundTab);
# backup from backend server 1, engine 10001
$ie.Navigate2("http://192.168.75.131:20001/DREEXPORTIDX?FileName=BU", $navOpenInBackgroundTab);
# backup from backend server 2, engine 20001
$ie.Visible = $true;

5. After the backups have been taken, start M-Files Indexing Services (MFIndexingManager).

2. Restoring an index from the backup

2.1 Process

1. Stop the M-Files Indexing Services (MFIndexingManager). Note that this stops all indexing so new material will not be added during the stop.

2. Wait until ALL indexing queues are empty.

- IDOL indexing queue location in each engine (example): I:\Single24\IDOLServer\IDOL\content\status
- M-Files indexing queue location (example): C:\Program Files\M-Files\Server Vaults\Sample Vault\FileData\FileData\Temp\FIX\C_B91D75EF-A924-4583-A7BA-2A8F3C5614C3)

3. Clear the whole index by opening browser in the frontend server and using following command.

http://[frontend server IP]:[DIH/DAH port]/DREINITIAL?

Example: http://192.168.75.128:19001/DREINITIAL?

4. Restore the index by adding it to the each engine and Daily (if exists) from the corresponding backup. Open the browser in the frontend server and using following command. By default, the file is searched from the engine's folder I:\Single24\IDOLServer\IDOL\content.

http://[backend server IP address]:[backend engine port]/DREADD?[optional backup file path and filename]-0.idx.gz&CreateDatabase=True&DREDbName=[index name from e.g. C:\Program Files\M-Files\Server Vaults\Indexes\Combined\Autonomy_IDOL_DB.txt]

Example: http://192.168.75.130:10001/DREADD?BU-0.idx.gz&CreateDatabase=True&DREDbName=SomeUniqueIndexName

You can use a Power Shell script like below to run multiple restores simultaneously.

$navOpenInBackgroundTab = 0x1000;
$ie = new-object -com InternetExplorer.Application
#$ie.Navigate2("http://192.168.75.128:9001/DREADD? BU-0.idx.gz ");
# restore Daily
$ie.Navigate2("http://192.168.75.130:10001/DREADD?BU-0.idx.gz", $navOpenInBackgroundTab);
#restore backend server 1, engine 10001
$ie.Navigate2("http://192.168.75.131:20001/DREADD?BU-0.idx.gz", $navOpenInBackgroundTab);
# restore backend server 2, engine 20001
$ie.Visible = $true;

5. Check the number of documents by using MFAutonomy console and action getstatus against DIH/DAH

6. Restore IndexMlog and IndexFlog (overwrite). They are located in M-Files Servers index folder, usually at C:\Program Files\M-Files\Server Vaults\Indexes\Combined\M-Files.

7. Start M-Files Indexing Services (MFIndexingManager).

3. Using Combined Indexing Mode (M-Files 12.0.6622.0 >)

Note! Combined Indexing is available from M-Files 2018 version 12.0.6622.0.

By default, M-Files does the initial indexing by first indexing all the metadata and then the FileData. This is because the metadata is much faster to index than FileData, where the content of the file has to be extracted and analyzed for indexing. The idea of this approach is that when the metadata indexing finishes relatively quickly, we have at least something about the whole material to search from.

The downside of the default mode is that that the sync between metadata and the corresponding FileData is more sensitive to break if something goes wrong during the indexing. This is because there night be considerable gap between the indexed metadata and its FileData counterpart.

The combined indexing tries to cope that by indexing at both the same time, first MetaData and then the corresponding FileData. This way there are much better changes to keep object integrity intact if something happens during the indexing. The downside of the combined indexing is that the whole material has to be indexed before we have a "whole" index. Usually this is not an issue, since dtSearch is used as a primary search engine until the IDOL index is complete.

To enable the combined indexing, use the following registry setting in M-Files server:

Path: HKLM\SOFTWARE\Motive\M-Files\[version]\Server\MFServer\VaultOptions\[vault GUID]\Indices\[used index, usually C_B91D75EF-A924-4583-A7BA-2A8F3C5614C3]
Key name: EnableCombinedOutput
Key type: DWORD
Value: 1

 

(0 vote(s))
This article was helpful
This article was not helpful

Comments (0)
Post a new comment
 
 
Full Name:
Email:
Comments:
CAPTCHA Verification 
 
Please enter the text you see in the image into the textbox below. This is required to prevent automated registrations and form submissions.

Help Desk Software by Kayako case