When iManage released the IDOL indexer into the legal community, they were being very cautious about ensuring proper performance on this platform.  They require that you fill out an RFI document with information about your environment and submit it to them for examination.  They will then give you sizing recommendations to outline hardware specifications for proper performance of IDOL in your environment.  While this information is very useful, I’ve watched many of my clients mistakenly use that document, with no additional review, as the template for building out their servers.

At this point, you are probably asking “Why would I need to modify sizing recommendations from the vendor?”  The answer is that the recommendations that are provided from iManage are based solely upon running IDOL.  They do not take into consideration any of the overhead of the OS nor the ability to backup the IDOL collections.

RAM and the OS

The RAM requirements set forth in the sizing recommendations are for running IDOL only.  If they specify 28 GB of RAM, that is what is needed to manage the overhead of IDOL.  But what about the OS?  It still needs a good memory footprint to run as well.  As a general rule, I add between 2 and 4 GB of RAM on top of any sizing recommendation to allow for this variance.  This assumes that you are using Windows Server 2008 R2 and the official memory guidelines for this platform can be found here.  If it is a smaller site (1 Million documents or less) you can add 2 GB of RAM, but push for 4 GB if it is larger.

Backups

The sizing recommendations lay out your disk arrangement for the deployment of IDOL.  However, they do not specify any amount of space needed for backups.  Furthermore, backups must be stored on a disk local to the Indexer (presented from SAN or Direct Attached Storage) as you cannot backup the collections to a location on the network.  For this reason, you must add additional disk space to your indexer to house backups.

I usually allow 1/3 of the collection size for backups assuming compression is on.  I also use the maximum size of the content engine drive in my calculations as well (Active Content is not backed up).  If we look at this mathematically it would be ((X * Y) * 0.33) Where X = The size of a single content engine and Y = The number of content engines.  If a sizing recommendation specifies 4 content engines, each 100 GB, then we would need 132 GB of space for backups ((100 * 4) * 0.33)

So where do we put these backups? You have several options.  If you are SAN attached, the first thought is to present another drive(s) to the machine to host the backups.  However, this is an expensive option for something that is just housing backups.  Another option would be to put in a Direct Attached Storage (DAS) disk.  Since this is just a temporary location for these backups (and you’re likely copying them off to some other location, right?) you do not need to worry about RAID and other considerations for this disk.  I’m sure there are other options out there, but these seem to be the most common.  Just remember that you want to maintain a high I/O rate on the disk or else the backups will take longer.

Hopefully this gives you a little guidance on what to look for and how to balance out your sizing recommendation document to properly deploy IDOL.

So you’ve decided to use WorkSite Web as your Extranet?  This makes sense as you can control the content in one place, rather than publishing multiple copies to multiple locations.  Anyone that’s been through a litigation hold can appreciate this.  It also makes managing the current version of the official record much easier to maintain.  However, before you prop up a WorkSite Web Server and expose it to the world, take some time to consider the security ramifications.

I try to think of security in two contexts, system authentication and content access.  System authentication is how you gain access to the hardware or system that the content resides in.  Content access defines the scope of what content you can see within that system.  I try to rate products by scoring them against both of these areas and see where they end up.  I rate these on a low, medium, high scale.  A low value in System authentication means there is poor security access control to the system, while a low value in content access means that there is low security around content access.

While WorkSite Web does allow for some basic IIS Authentication to happen, it still relies on WorkSite Trusted Login.  The challenge with this is that it is not a true Windows AD challenge.  You can turn on Kerberos Authentication for WorkSite which improves this dramatically, but in my experience the majority of customers have not enabled this option.  Because of this, I’d score native WorkSite Web a “Medium” for system authentication.

Another challenge is that WorkSite Web is a full-feature client.  This makes sense for internal users, but you may not want to hand over the entire functionality set of WorkSite to external users.  Once a user authenticates into the system, they are able to see, browse, and search for anything they have access (or accidentally have access) to.  I’d score native WorkSite Web with “low” for content access.

So what can we do to improve our security scores?  Rather than deploying native WorkSite Web, consider deploying the WorkSite Web SharePoint Web Parts.  The WorkSite Web SharePoint Web Parts are a solution from Autonomy iManage that allow you to implement Web Parts on your SharePoint 2007/2010 sites that plug-in to your existing document management environment.  The credentials of your SharePoint session are passed through to iManage for trusted login.  What I like about this solution is it allows you to begin to wrap your Extranet solution with increasing levels of security.  In addition, you can use some of the excellent collaborative functions of SharePoint in conjunction with document access.

By wrapping SharePoint around the existing WorkSite functionality, you can begin to increase the level of security dramatically.  SharePoint will force a Kerberos / AD authentication check every time the user tries to access the page.  You can use forms / claims based authentication, but I recommend using only AD credentials for external users (external domain, read only domain controller, etc…) specifically for security access.  From here, you can begin to wrap other layers of security around SharePoint as well.  A great example would be to stand a netscaler or other device in front of this in a DMZ.  In this configuration, I’d raise the system authentication score to a “High”.

Another great feature of this product is that there are multiple web parts you can use for almost any level of access.  You can have a very granular lens on what content is shared with the system.  For example, the screenshot above shows a web part that grants the full client functionality of WorkSite.  But what if I only want to share a specific WorkSpace or Folder?  You can do that.  There is a control where you can pick what WorkSpace or Folder to show and then the page will look like this:

The WebParts you can pick from are staggering.  They really did a great job of giving you the option to lock down exactly what is displayed on the site.  Here is a list of the items:

With this level of granularity, the score for content access moves to a “High”.

Hopefully this gives you a new perspective on how to use WorkSite for your Extranet solution or to harden the solution you already have in place.

Today we are looking at an alternative way to backup the IDOL content engines. Using Task Scheduler and Powershell, we can add some useful functionality into our backup routines.

The “out of the box” backup method requires you to set a section in each config file with various parameters. This method works well but it does present some challenges around planned interruptions and reporting. What if you want to skip the backups this weekend because of a maintenance window? What if you want a log file of just the backups or an email notification upon completion / failure? In addition to the backups, you also need to copy .cfg and .db files. Let’s make things a little easier and combine all of these into one step!

If you’ve never used PowerShell before, it can be overwhelming at first. However, after some quick study, I think you’ll quickly find that it is capable of so much that you’ll soon be using it for most of your administration and automation tasks (yes…really…). If you’ve never run PowerShell on the IDOL server(s) before, you’ll need to open it up and set the execution policy to unrestricted. This allows scripts to be run on that machine from now on. The syntax is: Set-ExecutionPolicy Unrestricted -Scope LocalMachine

Now, let’s breakdown the actual script that we will use to perform the backups. This script assumes that we will be backing up 3 content engines. It also assumes that we are backing up every week, and only keeping the latest version of the backup since index time is so quick. The completed script looks like this:

#
# PS Script to Backup IDOL
# Created By JB Trexler
# Created On: 11/4/2011
#
# Description:
# This was written to backup the IDOL Config and database files.
# It will also launch the command to backup the collections via the web browser (silently)
#
# Define Directory Variables
 $IDOLDeployDir = "E:\Program Files\Autonomy\Indexer"
 $ConfigBAKDir = "J:\IDOLBAK\ConfigFiles"
 $IDOLCEBakDir1 = "J:\IDOLBAK\CE1"
 $IDOLCEBakDir2 = "J:\IDOLBAK\CE2"
 $IDOLCEBakDir3 = "J:\IDOLBAK\CE3"
# Declare IE as an object for each content engine that you need to backup.
 $ie1 = new-object -com internetexplorer.application
 $ie2 = new-object -com internetexplorer.application
 $ie3 = new-object -com internetexplorer.application
# Copy the Config and db files
 Copy-Item $IDOLDeployDir\*.cfg $ConfigBAKDir -recurse
 Copy-Item $IDOLDeployDir\*.db $ConfigBAKDir -recurse
# Remove Existing Backups
 Remove-Item $IDOLCEBakDir1\* -recurse
 Remove-Item $IDOLCEBakDir2\* -recurse
 Remove-Item $IDOLCEBakDir3\* -recurse
# Silently Launch IE to Backup the Directories
 $ie1.navigate("http://localhost:11001/DREBackup?"+$IDOLCEBakDir1)
 $ie1.visible = $false
 $ie2.navigate("http://localhost:12001/DREBackup?"+$IDOLCEBakDir2)
 $ie2.visible = $false
 $ie3.navigate("http://localhost:13001/DREBackup?"+$IDOLCEBakDir3)
 $ie3.visible = $false
# All Done
 Write-Host "Backups Issued.  Monitor the index log file of each content engine for completion status."

Save that file as IDOLBackups.ps1

The next Script is a variation on the first. This is designed to run the DRECOMPACT operation as a scheduled task. The completed script looks like this:

#
# PS Script to Compact IDOL
# Created By JB Trexler
# Created On: 11/4/2011
#
# Description:
# This was written to Compact the IDOL Content Engines.
# It will launch the command to compact the collections via the web browser (silently)

# Declare IE as an object for each content engine that you need to backup
$ie1 = new-object -com internetexplorer.application
$ie2 = new-object -com internetexplorer.application
$ie3 = new-object -com internetexplorer.application

# Silently Launch IE to Backup the Directories
$ie1.navigate("http://localhost:11001/DRECOMPACT")
$ie1.visible = $false
$ie2.navigate("http://localhost:12001/DRECOMPACT")
$ie2.visible = $false
$ie3.navigate("http://localhost:13001/DRECOMPACT")
$ie3.visible = $false

# All Done
Write-Host "DRECOMPACT Issued. Monitor the index log file of each content engine for completion status."

Save that file as IDOLCompact.ps1

From here, you can Schedule a Task to run both of these jobs in Task Scheduler. Keep in mind that the DRECOMPACT operation can take a VERY long time to run (several hours). As such, I like to schedule the DRECOMPACT and the DREBackups 24 hours apart. Depending on the environment, I’ll run the DRECOMPACT monthly and the DREBackups weekly.

Create a New Task:

Set the Schedule:

Set the Action as “Start a Program”. The command is C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe and the Arguments are -noninteractive -nologo J:\control\IDOLBackups.ps1 Swap your location of the IDOLBackups.ps1 file.

Repeat this process for the DRECOMPACT Job.

Since these are now scheduled tasks in Windows, you have more flexibility in doing things with these jobs. You can disable them temporarily if needed, send an email on completion / failure, log to the Windows Event Log and more. I hope this gives you some flexibility in your backup routine

Any WorkSite administrator has most likely been in a situation where a hardware failure or hardware upgade requires changing your configuration settings, then stopping and starting your services for this to take effect.  By proper use of DNS Aliases, you can minimize the amount of user outages and start to create a platform that is “fluid”.

Let’s examine the connection points in a simple WorkSite enviornment.  A user connects to a DMS.  This DMS connects to a SQL Server(s) to get database(s) information, a file server and a full text indexer.  Let’s say that these servers are named as follows:

If you used the actual hostname for each connection, any hardware failure or equipment replacement would require a reconfiguration of the services and a potential outage to the users when stopping and starting the services.  Instead, consider creating a virtual name, or variable, for the connection information:

This gets us half of the way there.  The rest of the work is done in DNS.  There are several types of DNS entries that can be created.  For the purposes of this post we will focus on two, A records and CNAME records.  An A record is something that matches a machine name with an IP address.  These are the most common records created in DNS and is what is created for every machine and server on a domain.  A CNAME record serves as an alias or shortcut to another machine.

Typically A records are used for the DMS as you’ll likely have multiple and want to reference them in some sort of Round Robin.  In the above example, you would create two A records, each called FirmDMS and point each record to a unique IP of a DMS.  CNAMEs come in handy when you have multiple machines that serve the same role.  In the above example, we have FirmIDX01 and FrimIDX02. Let’s say these servers are independant servers indexing side by side.  The DMS server points to an active indexer, but needs to be able to fail over to the other indexer should the need arise.  In this case, we create a CNAME called FirmIDX and point it to FirmIDX01 (the active node).

The third and most important element in this process is the Time to live (TTL) value for these records.  The TTL value determines how often a client machine will check in with DNS to see if the record has changed.  By default, the TLL for all entries is 1 day (24 hours).  If we don’t modify this value, that means that the server will cache the DNS entry for 24 hours before checking back in with DNS to see if there is a new value.  This timeframe is not acceptable when you need to fail something over and have it take effect immediately.

What the TTL should actually be is a matter of preference.  I often ask “How long do you want to wait before the servers pick up the new connection information in the event of an outage?”.  The value that I commonly recommend is one minute or less.  This means that when you change the value in DNS, the machines should pick it up one minute or less after it has been changed.  A common question I also get is “How do I change the TTL for just one or more DNS entries?”  This view is turned off by default in DNS.  If you select the “View” menu and check “Advanced”, you’ll be able to see and modify the TTL for each record.

With everything set properly, any changes to the hardware in the above scenario can be done in DNS, eliminating the need for an outage due to recycling the WorkSite Services.  Likewise, should an entire failover become necessary, this aleviates the complexity of managing multiple machines in a production and DR site by keeping the settings consistent. I hope this helps with your design!