Summary:

An exception may be raised when synchronising tasks. This error will prevent subsequent tasks from being synched in affected mailboxes.

Symptoms:

crm.log:

ERROR [http-apr-11009-exec-9]: 18-Jun-2013 11:52:26.117 Due to processing errors UUID = 5d7d1e5f-1207-49be-b59b-6aedef127cdd will be removed from SyncSource response feed.
java.util.ConcurrentModificationException
at java.util.ArrayList$Itr.checkForComodification(Unknown Source)
at java.util.ArrayList$Itr.next(Unknown Source)
at com.sage.crm.exchange.util.LoggerSyncExchange.getExchangeLog(LoggerSyncExchange.java:164)
at com.sage.crm.exchange.util.LoggerSyncExchange.info(LoggerSyncExchange.java:97)
at com.sage.crm.exchange.endpoint.SyncSourceManager.createPayloadForEntity(SyncSourceManager.java:1623)
at com.sage.crm.exchange.endpoint.SyncSourceManager.generatePayload(SyncSourceManager.java:339)
at com.sage.crm.exchange.endpoint.SyncSourceManager$$FastClassByCGLIB$$5dfba5a4.invoke(<generated>)
at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:191)
at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:688)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:110)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:621)
at com.sage.crm.exchange.endpoint.SyncSourceManagerExchange$$EnhancerByCGLIB$$9a77ac94.generatePayload(<generated>)
at com.sage.crm.exchange.service.SyncSourceSDataStore.handleSubsequentRequest(SyncSourceSDataStore.java:540)
at com.sage.crm.exchange.service.SyncSourceSDataStore.generateContent(SyncSourceSDataStore.java:295)
at com.sage.crm.exchange.service.SyncSourceSDataStore.getData(SyncSourceSDataStore.java:199)
at com.sage.crm.exchange.controller.SyncController.handleSyncRequest(SyncController.java:44)
at com.sage.crm.exchange.controller.SyncSourceController.getSyncSource(SyncSourceController.java:57)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:212)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:126)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:96)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:617)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:578)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:80)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:900)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:827)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:778)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at com.sage.crm.web.filters.ClearCurrentThreadLocalsFilter.doFilter(ClearCurrentThreadLocalsFilter.java:45)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:311)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:116)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:101)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:201)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at com.sage.crm.core.security.SIDAuthenticationFilter.doFilter(SIDAuthenticationFilter.java:90)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:173)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:259)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.springframework.orm.hibernate4.support.OpenSessionInViewFilter.doFilterInternal(OpenSessionInViewFilter.java:119)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:225)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:927)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:999)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:565)
at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(AprEndpoint.java:1812)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

ERROR [http-apr-11009-exec-4]: 18-Jun-2013 11:53:04.538 java.util.ConcurrentModificationException
ERROR [http-apr-11009-exec-4]: 18-Jun-2013 11:53:04.577 com.sage.scrm.model.error.SageSynchronizationException: java.util.ConcurrentModificationException


exchangesyncenginestacktrace.log:

2013-06-18/11:53:10.695/EDT [SyncEngine thread] ERROR com.sage.scrm.syncengine.exchange.engine.SynchronisationThread.run com.sage.scrm.syncengine.core.engine.error.SageSyncEngineException: com.sage.scrm.syncengine.exchange.engine.model.error.SageSDataEndpointException: com.sage.scrm.syncengine.exchange.engine.model.error.SageSDataEndpointException: com.sage.crm.httpconsumer.error.SageHttpConsumerException: POST on http://CRM-SERVER/sdata/crmj/crmExchange/-/tasks/$syncResults?trackingId=de162d54-dce2-4d3f-8df1-386e1084670d failed: 500
com.sage.scrm.syncengine.core.engine.error.SageSyncEngineException: com.sage.scrm.syncengine.exchange.engine.model.error.SageSDataEndpointException: com.sage.scrm.syncengine.exchange.engine.model.error.SageSDataEndpointException: com.sage.crm.httpconsumer.error.SageHttpConsumerException: POST on http://CRM-SERVER/sdata/crmj/crmExchange/-/tasks/$syncResults?trackingId=de162d54-dce2-4d3f-8df1-386e1084670d failed: 500
at com.sage.scrm.syncengine.exchange.engine.SynchronisationProcess.execute(SynchronisationProcess.java:202)
at com.sage.scrm.syncengine.exchange.engine.SynchronisationThread.synchroniseResourceOneWay(SynchronisationThread.java:296)
at com.sage.scrm.syncengine.exchange.engine.SynchronisationThread.synchroniseResourcesOneWay(SynchronisationThread.java:281)
at com.sage.scrm.syncengine.exchange.engine.SynchronisationThread.run(SynchronisationThread.java:191)
at java.lang.Thread.run(Unknown Source)
Caused by: com.sage.scrm.syncengine.exchange.engine.model.error.SageSDataEndpointException: com.sage.scrm.syncengine.exchange.engine.model.error.SageSDataEndpointException: com.sage.crm.httpconsumer.error.SageHttpConsumerException: POST on http://CRM-SERVER/sdata/crmj/crmExchange/-/tasks/$syncResults?trackingId=de162d54-dce2-4d3f-8df1-386e1084670d failed: 500
at com.sage.scrm.syncengine.exchange.engine.SynchronisationProcess$SourceEndpointThread.run(SynchronisationProcess.java:296)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
... 1 more
Caused by: com.sage.scrm.syncengine.exchange.engine.model.error.SageSDataEndpointException: com.sage.crm.httpconsumer.error.SageHttpConsumerException: POST on http://CRM-SERVER/sdata/crmj/crmExchange/-/tasks/$syncResults?trackingId=de162d54-dce2-4d3f-8df1-386e1084670d failed: 500
at com.sage.scrm.syncengine.exchange.sdata.SCRMEndpoint.sendSyncTargetResponse(SCRMEndpoint.java:179)
at com.sage.scrm.syncengine.exchange.engine.SynchronisationProcess$SourceEndpointThread.run(SynchronisationProcess.java:279)
... 6 more
Caused by: com.sage.crm.httpconsumer.error.SageHttpConsumerException: POST on http://CRM-SERVER/sdata/crmj/crmExchange/-/tasks/$syncResults?trackingId=de162d54-dce2-4d3f-8df1-386e1084670d failed: 500
at com.sage.crm.httpconsumer.HttpConsumer.doPostPut(HttpConsumer.java:982)
at com.sage.crm.httpconsumer.HttpConsumer.doPost(HttpConsumer.java:761)
at com.sage.crm.httpconsumer.HttpConsumer.doPost(HttpConsumer.java:727)
at com.sage.scrm.syncengine.exchange.sdata.SCRMEndpoint.sendSyncTargetResponse(SCRMEndpoint.java:172)
... 7 more


exchangesynchronisation.log:

2013-06-18/11:53:10.695/EDT [SyncEngine thread] ERROR com.sage.scrm.syncengine.exchange.engine.SynchronisationThread.run com.sage.scrm.syncengine.core.engine.error.SageSyncEngineException: com.sage.scrm.syncengine.exchange.engine.model.error.SageSDataEndpointException: com.sage.scrm.syncengine.exchange.engine.model.error.SageSDataEndpointException: com.sage.crm.httpconsumer.error.SageHttpConsumerException: POST on http://CRM-SERVER/sdata/crmj/crmExchange/-/tasks/$syncResults?trackingId=de162d54-dce2-4d3f-8df1-386e1084670d failed: 500


exchangesynch.log:

ERROR [http-apr-11009-exec-2]: 14-Jun-2013 12:29:07.873 url: http://CRM-SERVER/sdata/crmj/crmExchange/-/tasks/$syncResults?trackingId=f496ed17-7d18-4efa-a74b-6cb4a2a80437, error: com.sage.crm.httpconsumer.error.SageHttpConsumerException: POST on http://CRM-SERVER/sdata/crmj/crmExchange/-/tasks/$syncResults?trackingId=f496ed17-7d18-4efa-a74b-6cb4a2a80437 failed: 500, action: ESTABLISHING_CONNECTION_CRM_ENDPOINT

Work around / Resolution:

The error was caused by corrupt items present on the Exchange side of the integration. Since the corrupt items were quite old, it was sufficient to remove entries on EcngSyncResource for the affected records.

In this instance, we removed all entries on EcngSyncResource, then ran a synch.