Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Share indexes among workspaces #2341

Merged
merged 7 commits into from
Jan 16, 2023
Merged

Share indexes among workspaces #2341

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
301f48a
Share indexes among workspaces
testforstephen Nov 22, 2022
3a08be5
Ask a read lock before copying local index
testforstephen Nov 23, 2022
e4906fd
Wait for index ready before copy & Cleanup unused local indexes
testforstephen Nov 29, 2022
e57ff86
Wait for index ready before copy
testforstephen Nov 29, 2022
b9dcb1e
trigger reindex when the external library contents has changed
testforstephen Dec 16, 2022
425fe2b
Add unit test
testforstephen Dec 16, 2022
1a0a3d4
Update TP to I20221215-1800 build
testforstephen Dec 16, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Ask a read lock before copying local index
  • Loading branch information
@testforstephen
testforstephen committed Dec 16, 2022
commit 3a08be5af43bc6db5c597830da52f6854b073d5c
60 changes: 39 additions & 21 deletions org.eclipse.jdt.ls.core/src/org/eclipse/jdt/ls/core/internal/handlers/IndexUtils.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,22 +29,27 @@
import org.eclipse.jdt.core.IJavaProject;
import org.eclipse.jdt.core.JavaCore;
import org.eclipse.jdt.core.JavaModelException;
import org.eclipse.jdt.internal.core.JavaProject;
import org.eclipse.jdt.internal.core.ClasspathEntry;
import org.eclipse.jdt.internal.core.JavaModelManager;
import org.eclipse.jdt.internal.core.JavaProject;
import org.eclipse.jdt.internal.core.index.FileIndexLocation;
import org.eclipse.jdt.internal.core.index.Index;
import org.eclipse.jdt.internal.core.index.IndexLocation;
import org.eclipse.jdt.internal.core.search.indexing.IndexManager;
import org.eclipse.jdt.internal.core.search.indexing.ReadWriteMonitor;
import org.eclipse.jdt.ls.core.internal.JavaLanguageServerPlugin;
import org.eclipse.jdt.ls.core.internal.ProjectUtils;

public class IndexUtils {
public static void copyIndexesToSharedLocation() {
IndexManager indexManager = JavaModelManager.getIndexManager();
// common index location for all workspaces
final String SHARED_INDEX_LOCATION = System.getProperty("jdt.core.sharedIndexLocation");
if (StringUtils.isBlank(SHARED_INDEX_LOCATION)) {
if (indexManager == null || StringUtils.isBlank(SHARED_INDEX_LOCATION)) {
return;
}

Set<IndexLocation> reusedIndexLocations = new HashSet<>();
Set<IndexLocation> resolvedIndexLocations = new HashSet<>();
IPath javaCoreStateLocation = JavaCore.getPlugin().getStateLocation();
for (IJavaProject javaProject : ProjectUtils.getJavaProjects()) {
try {
Expand All @@ -56,14 +61,12 @@ public static void copyIndexesToSharedLocation() {
continue;
}

IPath containerPath = entry.getPath();
IndexLocation localIndexLocation = getLocalIndexLocation(javaCoreStateLocation, containerPath);
IndexLocation sharedIndexLocation = IndexLocation.createIndexLocation(indexURL);
if (sharedIndexLocation.equals(localIndexLocation) || reusedIndexLocations.contains(localIndexLocation)) {
IndexLocation localIndexLocation = getLocalIndexLocation(javaCoreStateLocation, entry.getPath());
if (sharedIndexLocation.equals(localIndexLocation) || !resolvedIndexLocations.add(localIndexLocation)) {
continue;
}

reusedIndexLocations.add(localIndexLocation);
boolean needDeleteLocalIndex = false;
File localIndexFile = localIndexLocation.getIndexFile();
if (localIndexLocation.exists()) {
Expand All @@ -72,20 +75,37 @@ public static void copyIndexesToSharedLocation() {
continue;
}

File tmpFile = new File(sharedIndexFile.getParent(), sharedIndexFile.getName() + "." + System.currentTimeMillis());
try {
mkdirsFor(tmpFile);
Files.copy(localIndexFile.toPath(), tmpFile.toPath(), LinkOption.NOFOLLOW_LINKS, StandardCopyOption.REPLACE_EXISTING);
Files.deleteIfExists(sharedIndexFile.toPath());
needDeleteLocalIndex = tmpFile.renameTo(sharedIndexFile);
} catch (IOException e) {
JavaLanguageServerPlugin.logError(String.format("Failed to copy the local index %s to the shared index %s: %s",
localIndexFile.getName(), sharedIndexFile.getName(), e.getMessage()));
} finally {
if (indexManager.getIndex(sharedIndexLocation) != null) {
// current classpath entry is using the shared index, delete local index directly.
needDeleteLocalIndex = true;
} else {
Index localIndex = indexManager.getIndex(localIndexLocation);
if (localIndex == null) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might happen that after project imported, the index manager has not setup the index locations, then we skip a lot of local index.

Copy link
Contributor

@rgrunber rgrunber Nov 24, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the job to write out the indices is done in the background, so you can definitely have a case where for large libraries, they are still being written to disk after import. This is one of the reasons why in Eclipse (and likely VS Code) you can try to search after the project is imported and not get any results.

Would something like https://github.com/eclipse/eclipse.jdt.ls/blob/250ac4ff3d169ad56bf68bf6cb7bb48ef181be1f/org.eclipse.jdt.ls.core/src/org/eclipse/jdt/ls/core/internal/JobHelpers.java#L242 help ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

waitUntilIndexesReady seems to work quite well. It can ensure all indexes are generated before copying.

continue;
}

ReadWriteMonitor monitor = localIndex.monitor;
if (monitor == null) { // index got deleted since acquired
continue;
}

File tmpFile = new File(sharedIndexFile.getParent(), sharedIndexFile.getName() + "." + System.currentTimeMillis());
try {
Files.deleteIfExists(tmpFile.toPath());
monitor.enterRead();
mkdirsFor(tmpFile);
Files.copy(localIndexFile.toPath(), tmpFile.toPath(), LinkOption.NOFOLLOW_LINKS, StandardCopyOption.REPLACE_EXISTING);
Files.deleteIfExists(sharedIndexFile.toPath());
needDeleteLocalIndex = tmpFile.renameTo(sharedIndexFile);
} catch (IOException e) {
// do nothing
JavaLanguageServerPlugin.logError(String.format("Failed to copy the local index %s to the shared index %s: %s",
localIndexFile.getName(), sharedIndexFile.getName(), e.getMessage()));
} finally {
monitor.exitRead();
try {
Files.deleteIfExists(tmpFile.toPath());
} catch (IOException e) {
// do nothing
}
}
}
}
Expand Down Expand Up @@ -117,9 +137,7 @@ private static IndexLocation getLocalIndexLocation(IPath savedIndexPath, IPath c
}

private static void mkdirsFor(File destination) {
//does destination directory exist ?
if ( destination.getParentFile() != null && !destination.getParentFile().exists()) {
//noinspection ResultOfMethodCallIgnored
destination.getParentFile().mkdirs();
}
}
Expand Down
4 changes: 2 additions & 2 deletions org.eclipse.jdt.ls.core/src/org/eclipse/jdt/ls/core/internal/handlers/JDTLanguageServer.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -284,11 +284,11 @@ public IStatus run(IProgressMonitor monitor) {
// before send the service ready notification, make sure all bundles are synchronized
synchronizeBundles();

IndexUtils.copyIndexesToSharedLocation();

client.sendStatus(ServiceStatus.ServiceReady, "ServiceReady");
status = ServiceStatus.ServiceReady;
pm.projectsImported(monitor);

IndexUtils.copyIndexesToSharedLocation();
} catch (OperationCanceledException | CoreException e) {
logException(e.getMessage(), e);
return Status.CANCEL_STATUS;
Expand Down