Page 1 of 1

Truncated chunk

Posted: Wed May 29, 2019 10:54 am
by domlatham
We are regularly unable to access files with the following error:
Cannot open the specified file.
Truncated chunk ( expected size: 31868; actual size: 22074)
(numbers vary)

Under Details we get:
org.apache.http.TruncatedChunkException: Truncated chunk ( expected size: 31868; actual size: 22074)
at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:200)
at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:137)
at java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:238)
at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:158)
at java.util.zip.GZIPInputStream.read(GZIPInputStream.java:117)
at org.apache.http.client.entity.LazyDecompressingInputStream.read(LazyDecompressingInputStream.java:73)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at ro.sync.io.f.read(Unknown Source)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at ro.sync.basic.xml.encoding.EncodingDetectorImpl$EncodingErrorReportingReader.read(EncodingDetectorImpl.java:1492)
at ro.sync.exml.editor.xmleditor.prettyprint.c$1.read(Unknown Source)
at ro.sync.exml.editor.y.read(Unknown Source)
at java.io.BufferedReader.read1(BufferedReader.java:210)
at java.io.BufferedReader.read(BufferedReader.java:286)
at ro.sync.basic.io.RewindableReader.read(RewindableReader.java:230)
at ro.sync.basic.io.EmptyLineBufferedReader.fill(EmptyLineBufferedReader.java:119)
at ro.sync.basic.io.EmptyLineBufferedReader.readLine(EmptyLineBufferedReader.java:303)
at ro.sync.basic.io.EmptyLineBufferedReader.readLine(EmptyLineBufferedReader.java:373)
at ro.sync.basic.io.IOUtil.readWithLongLineDetection(IOUtil.java:615)
at ro.sync.basic.io.IOUtil.readWithLongLineDetection(IOUtil.java:582)
at ro.sync.basic.io.IOUtil.readWithLongLineDetection(IOUtil.java:569)
at ro.sync.exml.editor.pagetext.f.ibm(Unknown Source)
at ro.sync.exml.editor.pagetext.f.loadPageData(Unknown Source)
at ro.sync.exml.editor.q.loadPage(Unknown Source)
at ro.sync.exml.editor.xmleditor.e.j.loadPage(Unknown Source)
at ro.sync.exml.editor.xmleditor.e.c.loadPage(Unknown Source)
at ro.sync.exml.editor.wc.readDocument(Unknown Source)
at ro.sync.exml.editor.validate.r.readDocument(Unknown Source)
at ro.sync.exml.editor.xmleditor.rb.readDocument(Unknown Source)
at ro.sync.exml.editor.me.y(Unknown Source)
at ro.sync.exml.editor.me.hb(Unknown Source)
at ro.sync.exml.editor.wc.open(Unknown Source)
at ro.sync.exml.editor.xmleditor.rb.open(Unknown Source)
at ro.sync.exml.editor.ae.ati(Unknown Source)
at ro.sync.exml.editor.ae.open(Unknown Source)
at ro.sync.exml.k.v.c(Unknown Source)
at ro.sync.exml.k.v.d(Unknown Source)
at ro.sync.exml.k.v.e(Unknown Source)
at ro.sync.exml.editor.s.bb$36.b(Unknown Source)
at ro.sync.db.nxd.c.jk(Unknown Source)
at ro.sync.db.nxd.c$4.bli(Unknown Source)
at ro.sync.db.f.z.byp(Unknown Source)
at ro.sync.db.f.z$1.dxd(Unknown Source)
at ro.sync.ui.application.mb.run(Unknown Source)

We are using Oxygen 21 connecting to eXist 4.6.1 using WebDAV.
I appreciate that is a lot of links in the chain but as the error is being raised by Oxygen I was hoping you could help shed some light on what is going on.

With many thanks

Re: Truncated chunk

Posted: Thu May 30, 2019 9:18 am
by Radu
Hi,

Oxygen and the Exist server are communicating using HTTP. Sometimes when using HTTP the server may indicate that it can send the content as zipped in gzip format (as it minimizes the bandwidth). This is the case here, Oxygen receives zipped content from the server, its HTTP libraries try to unzip it and fail to do that.

Did you obtain a similar error with previous Exist server versions?
Do you have an HTTP proxy web server set up in Oxygen (Preferences->Network Connection Settings / Proxy)? If so can you try to connect to the server bypassing the HTTP proxy?

Regards,
Radu

Re: Truncated chunk

Posted: Thu May 30, 2019 2:31 pm
by domlatham
Dear Radu,

Thanks for looking into this.
Did you obtain a similar error with previous Exist server versions?
We had no problem with our previous version of exist 3.4.1. The issues started on upgrading to 4.6.1. We have now upgraded to 4.7.0 but the issue persists.
Do you have an HTTP proxy web server set up in Oxygen (Preferences->Network Connection Settings / Proxy)? If so can you try to connect to the server bypassing the HTTP proxy?
No. It is set to "Use systems settings". I tried "Direct connection" but that didn't help.

With best wishes,

Dominic

Re: Truncated chunk

Posted: Fri May 31, 2019 10:51 am
by Radu
Hi Dominic,

I see you already asked on the Exist users list about this? I revived your thread, maybe someone else has seen this problem.
There could be two potential problems here:

1) At some point your network settings changed (maybe some internal routing mechanism started discarded HTTP packages which are over a certain limit). If this would be true then you could also switch to Exist 3.4.1 and the problem could be reproduced even with that server.

2) There is a bug (or a change) which was added in Exist 4.6.1 which makes the problem sometimes appear.

Do you consistently obtain this error for a certain XML file? If so, what exactly do you do when the error is obtained? Double click the XML file in the Data Source Explorer view? Is it a very large XML file? Can you always reproduce the problem for XML files over a certain file size?

Regards,
Radu

Re: Truncated chunk

Posted: Fri May 31, 2019 2:28 pm
by Contra8
We are facing the same problem here. We have a running system eXist-db 2.2 and Oxygen 20. Works fine.

This week, I installed an eXist 4.6, restored the content and users from 2.2 database. The problem occurs with bigger XML files, starting at a file sizes of around 950 kB.

Image

Detailed error message:

org.apache.http.TruncatedChunkException: Truncated chunk ( expected size: 30726; actual size: 15856)
at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:200)
at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:137)
at java.util.zip.InflaterInputStream.fill(Unknown Source)
at java.util.zip.InflaterInputStream.read(Unknown Source)
at java.util.zip.GZIPInputStream.read(Unknown Source)
at org.apache.http.client.entity.LazyDecompressingInputStream.read(LazyDecompressingInputStream.java:73)
at java.io.FilterInputStream.read(Unknown Source)
at ro.sync.io.f.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read1(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
at sun.nio.cs.StreamDecoder.readBytes(Unknown Source)
at sun.nio.cs.StreamDecoder.implRead(Unknown Source)
at sun.nio.cs.StreamDecoder.read(Unknown Source)
at java.io.InputStreamReader.read(Unknown Source)
at ro.sync.basic.xml.encoding.EncodingDetectorImpl$EncodingErrorReportingReader.read(EncodingDetectorImpl.java:1494)
at ro.sync.exml.editor.xmleditor.prettyprint.c$1.read(Unknown Source)
at ro.sync.exml.editor.x.read(Unknown Source)
at java.io.BufferedReader.read1(Unknown Source)
at java.io.BufferedReader.read(Unknown Source)
at ro.sync.basic.io.RewindableReader.read(RewindableReader.java:230)
at ro.sync.basic.io.EmptyLineBufferedReader.fill(EmptyLineBufferedReader.java:164)
at ro.sync.basic.io.EmptyLineBufferedReader.readLine(EmptyLineBufferedReader.java:348)
at ro.sync.basic.io.EmptyLineBufferedReader.readLine(EmptyLineBufferedReader.java:418)
at ro.sync.basic.io.IOUtil.readWithLongLineDetection(IOUtil.java:660)
at ro.sync.basic.io.IOUtil.readWithLongLineDetection(IOUtil.java:627)
at ro.sync.basic.io.IOUtil.readWithLongLineDetection(IOUtil.java:614)
at ro.sync.exml.editor.pagetext.f.iij(Unknown Source)
at ro.sync.exml.editor.pagetext.f.pvh(Unknown Source)
at ro.sync.exml.editor.p.ndg(Unknown Source)
at ro.sync.exml.editor.xmleditor.e.j.ndg(Unknown Source)
at ro.sync.exml.editor.xmleditor.e.c.ndg(Unknown Source)
at ro.sync.exml.editor.uc.readDocument(Unknown Source)
at ro.sync.exml.editor.validate.q.readDocument(Unknown Source)
at ro.sync.exml.editor.xmleditor.qb.readDocument(Unknown Source)
at ro.sync.exml.editor.ke.x(Unknown Source)
at ro.sync.exml.editor.ke.fb(Unknown Source)
at ro.sync.exml.editor.uc.open(Unknown Source)
at ro.sync.exml.editor.xmleditor.qb.open(Unknown Source)
at ro.sync.exml.editor.yd.sgk(Unknown Source)
at ro.sync.exml.editor.yd.open(Unknown Source)
at ro.sync.exml.k.u.c(Unknown Source)
at ro.sync.exml.k.u.d(Unknown Source)
at ro.sync.exml.k.u.e(Unknown Source)
at ro.sync.exml.editor.s.ab$38.b(Unknown Source)
at ro.sync.db.nxd.c.am(Unknown Source)
at ro.sync.db.nxd.c$4.qqh(Unknown Source)
at ro.sync.db.f.bb.xup(Unknown Source)
at ro.sync.db.f.bb$1.cud(Unknown Source)
at ro.sync.ui.application.nb.run(Unknown Source)

Re: Truncated chunk

Posted: Fri May 31, 2019 3:29 pm
by Radu
Hi,

Thanks for joining in on this. Does the problem occur consistently? Does it occur consistently for any XML file over 1MB or only for some of them? If it occurs only for some XML files can I perhaps get a sample XML file to test on my side (support@oxygenxml.com)?
What I managed to test on my side were two Exist versions (version 4.6.0 and 4.7.0), in both I uploaded three XML docs, all of them over 1 MB, the largest has 3MBs. But I am able to open the XML files from the server in Oxygen 21.1 and work with them. I am unable to find a place from where to download Exist 4.6.1 so I am unable to test with it.

Regards,
Radu

Re: Truncated chunk

Posted: Fri May 31, 2019 4:53 pm
by Contra8
This problem occurs permanently for every file over a certain size which is around 1 MB. I will send you an example file to the given e-mail address.

I got eXist 4.6.1 from JFrog: https://bintray.com/existdb/releases/exist/4.6.1/view

Maybe I should give it a try with 4.7.0 and see if it works there ...

Re: Truncated chunk

Posted: Fri May 31, 2019 10:24 pm
by DannesW
Please use the REST interface instead of the webDAV interface, as the content-length header of GET on this interface might not give the accurate value. We do this for performance optimization, MOST webdav clients can deal with this.

There are a few exist-db settings you can change. These are documented in the eXist-db documentation (sorry don;t have a server available at this moment)

Re: Truncated chunk

Posted: Mon Jun 03, 2019 9:31 am
by Radu
Hi,

So indeed in Oxygen you can either:

1) Connect using Webdav to the Exist server by connecting a Webdav connection in the Preferences Data Sources page.
or
2) You can create an Exist data source using the "Create eXist-db XML connection" link in the same Preferences Data Sources page.

So Dannes' advice is for you to use the (2) approach.
I still cannot reproduce the problem using either the (1) or the (2) approach with Exist 4.7.0 and Oxygen 21.1 to open some 3MB sized files.

Update: tested with Exist 4.6.1 as well and still no luck reproducing the problem on my side using the over 1MB sample XML documents I have.
So having a sample XML document to work with would help.
But you should also try connecting to the Exist server using the (2) approach.

Regards,
Radu

Re: Truncated chunk

Posted: Mon Jun 03, 2019 11:53 am
by Radu
Hi,

Using Dominic's sample I managed to obtain this HTTP exception when opening the file from an Exist 4.6.1 file:

Code: Select all

org.apache.http.MalformedChunkCodingException: CRLF expected at end of chunk
	at org.apache.http.impl.io.ChunkedInputStream.getChunkSize(ChunkedInputStream.java:255)
	at org.apache.http.impl.io.ChunkedInputStream.nextChunk(ChunkedInputStream.java:227)
	at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:186)
	at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:137)
I only obtained the error once although I opened the XML document lots of times so strangely the error is not consistent, at least on my side.
I'm adding a link also to the post suggested by Joe Wicentowski as a workaround for the problem: https://github.com/eXist-db/exist/issue ... -435027483

Dominic added the parameter on his server so he'll probably report back if the problem seems to be fixed or not.

Regards,
Radu

Re: Truncated chunk

Posted: Tue Jun 04, 2019 6:07 pm
by Contra8
I tried now to use different versions of eXist but the problem stays the same with 4.7, 4.6.1, 3.*.

Dear Radu, I sent you an XML file that is affected via e-mail to support@ixygenxml.com. We created those files when working with eXist 2.2. Now after upgrading we face the problem described here permanently.

Re: Truncated chunk

Posted: Wed Jun 26, 2019 1:19 pm
by Radu
Hi,

Can you both confirm that if you uncheck the checbox "Lock WebDav Files on open" from the Preferences->"Network Connection Settings / HTTP(S)/WebDAV" page you no longer can reproduce the problem when opening the large XML documents via WebDav?
If so, I have a couple ideas about what to do in our code in order to work around this problem in a future Oxygen version.

Regards,
Radu

Re: Truncated chunk

Posted: Wed Jun 26, 2019 3:37 pm
by Contra8
Dear Radu,

sorry for replying late. I had to check several clients in my team.

You are right, when we uncheck the "Lock WebDAV files on open" in Oxygen preferences the problem disappears. Even the large files are opened immediately. No error message occurs.

I checked this with several Oxygen clients 20.0 on both Windows and Mac.

Interesting side note: I have a team member who runs Oxygen 18.0 on a Mac. On this machine, the described problem does not appear at all. Even when he checks "Lock WebDAV files on open" in Oxygen preferences, the large files open up immediately without any time-out or error message.

Kind regards
Christian

Re: Truncated chunk

Posted: Fri Jul 19, 2019 8:40 am
by Radu
Hi,

We made a minor bug fix release of Oxygen XML 21.1 and it should have the problem with opening large XML documents from Exist fixed:
https://www.oxygenxml.com/build_history.html#2019071807

Regards,
Radu