Category Archives: Exchange 2007

EternalBlue Exploit used to Deliver Remote Access Trojans

https://www.cyphort.com/eternalblue-exploit-actively-used-deliver-remote-access-trojans/

On May 16th we started experiencing mail flow issues on our unsupported Exchange 2007 environment hosted on Server 2003 OS’s. Mail was coming in over SMTP and being delivered to users as normal, however internal emails, and outgoing emails appeared to be going into $null. I could not see them in any Exchange logs, there was no trace of what was happening to them.

After some investigating, another engineer found two ports were blocked, 135 and 445 or RPC and MS Directory Services. The Windows firewall on all machines is disabled by default so we double checked that and moved on. First stop, our antivirus, we disabled and then completely uninstalled on what servers still had it running, wait how was AV missing on these servers? Next we checked our other fancy AI machine learning super advanced security application, Cylance, that was missing as well… hmm…

The workday had ended and myself and the other engineer were at home for the evening, I was VPN’ed in, determined to find what was blocking this traffic. I had seen a few event logs referencing IPsec and a GUID but I hadn’t thought much on those, I did a quick google search on IPsec rules and Server 2003, loaded up the IPsec Management Console and noticed there was a new rule created, last modified a few hours earlier, the rule was simply called ‘win’. Upon entering the rule I noticed there was a deny list, and sure enough, ports 135 and 445 were listed, unassigned the win IPsec rule, and mailflow started up again. I checked the 3 other 2003 servers, same rule, unassigned, mailflow issues resolved.

ipsec1

Also during investigation, in event logs, I found an MSI trying to install when the other admins were logging in. js.mykings.top was the name, and I found this in registry, scheduled tasks, and startup.

mykings

mykings1

mykings2

crypt1

itemvmapplet

New users not showing up in OAB but show in GAL in OWA

Event ID 9320
OALGen could not generate full details for some entries in the offline address list for address

Get-OfflineAddressBook – Identity “Default OAB” | fl
GUID: 2e91c924-5590-4013-94a2-0dc08fe9285e

I checked the OAB folder on the Generation Server, and the 2 distribution servers, and noticed one of the distribution servers had files that were not modified with todays date, but the generation server did, as did one of the distribution servers.  So it appears to be a replication issue.

D:Program FilesMicrosoftExchange ServerClientAccessOAB2e91c924-5590-4013-94a2-0dc08fe9285e

Our 2 HUB servers are set as the OAB Distribution points, but is showing files last modified 10/11/2016 while the other is today, 10/12/2016.  So there seems to be a replication issue between them.

I ran Update-FileDistrubtionService – Identity EXCH01 -Type OAB
This ran about 15 minutes and didn’t finish

On the server without the updated OAB lzx files I restarted the Microsoft Exchange File Distribution service

The lzx files last modified date then updated to today’s date.

From my Outlook i downloaded a new copy of the OAB from Send/Receive and tested by creating a new email and confirming the user missing was now available

OAB/GAL not updating with new users

-Create new OAB on generating server
-Navigate to C:Program FilesMicrosoftExchange ServerExchangeOAB and ensure there is a folder there -You may have to restart the Microsoft Exchange System Attendant service
-Go to Organization Configuration/Mailbox and Offline Address Book tab, make sure you are doing Web-Based distribution
-Navigate to your distribution servers to C:Program FilesMicrosoftExchange ServerClientAccessOAB and make sure the folder is there as well.  If it is not, restart the Microsoft Exchange File Distribution service and wait a bit.
-Go into IIS on the distribution servers, Application Pools, Recycle MSExchangeAutodiscoverAppPool
-In Outlook do an Test E-mail AutoConfiguration and ensure the OAB UID matches the folder name in the above directories

Unable to create storage group to many log files

As part of disaster recovery, I was trying to restore a database and 45,000 associated transaction logs.  I was getting the following error when trying to create a new storage group pointing to the restored data.
Error:
Failed to connect to the target server “MBX-05”. The exception message is “WMI exception occured on server ‘MBX-05’: Call cancelled “.

Exchange Management Shell command attempted:
new-StorageGroup -Server ‘MBX-05’ -Name ‘ Storage Group’ -LogFolderPath ‘F:ExchangeGroupsStorageGroupLogs’ -SystemFolderPath ‘F:ExchangeGroupsStorageGroupMailboxDatabase’

OST sync with mailbox after restore

If we restore Exchange 2007 data from 2 weeks ago and bring it back online, will Outlook clients sync the previous 2 weeks of email from the OST back to the Exchange mailbox after its restored from backup?

I seem to see split opinions on this, and no documentation to support either claim if it will or will not.

We just tested.  Outlook did not sync the previous 2 weeks of emails as I had thought back with Exchange.  However those 2 weeks of emails did stay put in the users Outlook so far…

relocating iSCSI volume with db/logs to a new server

We have an Exchange 2007 Mailbox server running on Server 2003. We want to build a Server 2008 box, and attach the Exchange iSCSI volume to the new server.

https://technet.microsoft.com/nl-nl/library/bb123954(v=exchg.80).aspx

As with previous versions of Microsoft Exchange, an upgrade of the operating system for an Exchange server results in the updating of the value for OS Version in the database header. This update triggers the rebuilding of internal database indexes. When using database portability to move a database from a Mailbox server running Windows Server 2003 to a Mailbox server running Windows Server 2008, the Extensible Storage Engine (ESE) detects the operating system upgrade and takes the following actions:

  • During the first database mount operation, all secondary indexes are discarded. A secondary index is used to provide a specific view of the mailbox data (for example, when messages in a mail folder are sorted using Outlook in Online mode). The database will not be mounted and available to clients until this initial operation is complete. The amount of time to complete the operation is largely dependent on the size of the database. The larger the database is, the longer the mount operation will take.
  • Secondary indexes will be rebuilt on-demand as Outlook users sort their views in Online mode. In environments with large or extremely large databases, the on-demand rebuilding of indexes will initially result in high processor and disk utilization.

Unmount databases on old Exchange MBX server
Stop Exchange Services on old MBX server
Disconnect iSCSI volume from old MBX server
Connect iSCSI volume to new MBX server
Mount iSCSI drive in Windows on new MBX server
Create Storage Groups and point to existing DB/Logs on iSCSI volume
Mount databases
Wait for indexing to take place before database remounts
Run PowerShell command to point mailboxes from old MBX to new MBX

Get-Mailbox -database “EXMBX1CORP Storage GroupMailbox Database” | Move-Mailbox -TargetDatabase “EXMBX3CORP Storage GroupMailbox Database”| -ConfigurationOnly

4.4.2 Connection Dropped Exchange 2007

Had an issue with a couple users trying to send to 1 specific domain that just seemed to have stopped working.  Exchange queue on our Edge server was showing the following error: 4.4.2 Connection Dropped

After several hours of research, and determining it was nothing on our side, I contacted the other sides IT team who reached out to rackspace who was hosting their email.

Logs on Edge on our side showed the following

C:Program FilesMicrosoftExchange ServerTransportRolesLogsConnectivity

2016-07-01T00:23:29.590Z,08D387F7316D769B,SMTP,domain.com,>,Established connection to XXX.XXX.XXX.XX1

2016-07-01T00:23:30.043Z,08D387F7316D769B,SMTP,domain.com,>,Established connection to XXX.XXX.XXX.XX2

2016-07-01T00:23:51.200Z,08D387F7316D769B,SMTP,domain.com,>,Failed connection to XXX.XXX.XXX.XX3 (0000274C)

2016-07-01T00:23:51.200Z,08D387F7316D769B,SMTP,domain.com,-,Retry

Why was it establishing a connection, but moving onto the next MX record still?

Ran MX toolbox

Ran Microsoft Exchange Connectivity Analyzer which showed it wasnt able to connect to the 3rd MX record

Questioned what the 3rd MX record was, and it was a stale on-prem email record that was never removed.  They updated DNS to remove this.  Let it propagate the internet

Ran a ipconfig /flushdns on Edge server

Cleared the messages on the Edge server in this domains queue, Suspended the Queue, Resumed it to clear it out, sent another email and it went through successfully.

Cause: Stale MX record on recipients side

Conclusion: Only explanation I can come up with is that the first 2 MX records for rackspace were unavailable and it tried and hung onto the 3rd stale MX record and kept trying to use that.