BT Farm Crash Fix
Symptoms
• Unable to assign users to farm for migration
• When you try to open CMTu it doesn’t open and then eventually you get error with “System.ArgumentNullException: Value cannot be null” in part of the error
• Syslog also shows the ‘Value cannot be null’ errors
• XML service error.
Caution: Please make sure this is the issue before proceeding as the fix is a bit destructive. You should attempt restarting IIS first.most of the tome restarting IIS will resolve issue.
Resolution
1. If you have attempted to open CMTu and haven’t closed it then use task manager to kill CMT.exe
2. Connect to all farm workstations. Wait for any active migrations to finish. You can also potentially pause the migrations (I usually do this for PP migrations) so I don’t have to wait.
3. Open Access: Start > Run > All Programs > Microsoft Office Access
4. In Access on right side pane there should be a quick link to CMT_Universal.mdb (the one that links to C:\Program Files\CMT Universal\CMT_XMLServer\App_Data\CMT_Universal.mdb), click that to open the database, when prompted at Security warning click Open
5. Double-click the ‘MigrationQueueTable’
6. Go File > Export… then change Save as type to ‘Text Files (*.txt; *.csv; *.tab; *.asc)’ and name it ‘MigrationQueueTable’. Export Text Wizard pops up, accept defaults and click Finish, click Ok.
7. Right click the top left most gray table box (left of migration_id column, above arrow) then click Delete Record to delete all records
8. Close Access, save if prompted
9. Restart IIS
10. Restart the farm workstations (using ‘Restart ICS workstations.bat’ on desktop in the Batch Files folder) – only do this if you didn’t pause some migrations, if you did then just restart CMT worker process on each machine that does not have a paused migration. After machines have restarted, reconnect to them.
11. Double-click the ‘CMT Universal’ shortcut on the desktop, click on Users… button (CMTu starting is an indicator that the issue is fixed but it’s also good to check that the ‘null’ error is not showing up in Kiwi Syslog)
12. Sort by Status column then walk the list of users starting with those that indicate ‘In progress’ (none of the values will be accurate and need to be reset). Note that it’s also best practice to take care of all the Exchange migrations before doing the Proofpoint migrations.
13. Right-click user, click Properties… and look for machine user was migrated on (Migration_Details/Migrated_by)
14. Go to the workstation indicated in CMT properties open c:\program files\cmt universal\logs, sort by date, most recent first and open the last log file for that user and check migration status.
15. Go back into CMTu and take one of the following actions:
a. If log on workstation reports user as ‘Migrated successfully’ then copy that into CMT properties ‘Migration_Status’ then click in an empty field then exit properties. User should show in CMTu as Migrated successfully
b. If log shows user ‘Migrated with errors’ or log is truncated with no summary then reset user and send back to same workstation and be quick. Note that once reset the EMM auto code may pick up the user and assign them and it may not be back to the same machine however I found that the message history of already migrated messages is by this time generally in the Access table so it’s generally not an issue. It is however a good idea to check that user doesn’t have duplicates by connecting to their mailbox after migration has completed. After re-setting, check to see if migration has stated on the migration workstations.
c. If CMTu properties doesn’t have a value for ‘Migrated_by’ then likely user was pending. Reset the user but don’t manually assign it for migration as the auto send code will do this for you.
d. If there is no log in the workstation for the user, reset the user in CMT. No further action is necessary.
16. Once you’ve gone through all the users migrations then you should restart CMTu interface since it will not update status till you close and re-open. After re-opening any users you marked as ‘Migrated successfully’ should be gone (picked up by EMM).
17. Monitor migration workstations for user progress
Symptoms
• Unable to assign users to farm for migration
• When you try to open CMTu it doesn’t open and then eventually you get error with “System.ArgumentNullException: Value cannot be null” in part of the error
• Syslog also shows the ‘Value cannot be null’ errors
• XML service error.
Caution: Please make sure this is the issue before proceeding as the fix is a bit destructive. You should attempt restarting IIS first.most of the tome restarting IIS will resolve issue.
Resolution
1. If you have attempted to open CMTu and haven’t closed it then use task manager to kill CMT.exe
2. Connect to all farm workstations. Wait for any active migrations to finish. You can also potentially pause the migrations (I usually do this for PP migrations) so I don’t have to wait.
3. Open Access: Start > Run > All Programs > Microsoft Office Access
4. In Access on right side pane there should be a quick link to CMT_Universal.mdb (the one that links to C:\Program Files\CMT Universal\CMT_XMLServer\App_Data\CMT_Universal.mdb), click that to open the database, when prompted at Security warning click Open
5. Double-click the ‘MigrationQueueTable’
6. Go File > Export… then change Save as type to ‘Text Files (*.txt; *.csv; *.tab; *.asc)’ and name it ‘MigrationQueueTable
7. Right click the top left most gray table box (left of migration_id column, above arrow) then click Delete Record to delete all records
8. Close Access, save if prompted
9. Restart IIS
10. Restart the farm workstations (using ‘Restart ICS workstations.bat’ on desktop in the Batch Files folder) – only do this if you didn’t pause some migrations, if you did then just restart CMT worker process on each machine that does not have a paused migration. After machines have restarted, reconnect to them.
11. Double-click the ‘CMT Universal’ shortcut on the desktop, click on Users… button (CMTu starting is an indicator that the issue is fixed but it’s also good to check that the ‘null’ error is not showing up in Kiwi Syslog)
12. Sort by Status column then walk the list of users starting with those that indicate ‘In progress’ (none of the values will be accurate and need to be reset). Note that it’s also best practice to take care of all the Exchange migrations before doing the Proofpoint migrations.
13. Right-click user, click Properties… and look for machine user was migrated on (Migration_Details/Migrated_by)
14. Go to the workstation indicated in CMT properties open c:\program files\cmt universal\logs, sort by date, most recent first and open the last log file for that user and check migration status.
15. Go back into CMTu and take one of the following actions:
a. If log on workstation reports user as ‘Migrated successfully’ then copy that into CMT properties ‘Migration_Status’ then click in an empty field then exit properties. User should show in CMTu as Migrated successfully
b. If log shows user ‘Migrated with errors’ or log is truncated with no summary then reset user and send back to same workstation and be quick. Note that once reset the EMM auto code may pick up the user and assign them and it may not be back to the same machine however I found that the message history of already migrated messages is by this time generally in the Access table so it’s generally not an issue. It is however a good idea to check that user doesn’t have duplicates by connecting to their mailbox after migration has completed. After re-setting, check to see if migration has stated on the migration workstations.
c. If CMTu properties doesn’t have a value for ‘Migrated_by’ then likely user was pending. Reset the user but don’t manually assign it for migration as the auto send code will do this for you.
d. If there is no log in the workstation for the user, reset the user in CMT. No further action is necessary.
16. Once you’ve gone through all the users migrations then you should restart CMTu interface since it will not update status till you close and re-open. After re-opening any users you marked as ‘Migrated successfully’ should be gone (picked up by EMM).
17. Monitor migration workstations for user progress
